Web-Scale Semantic Product Search with Large Language Models

Abstract Dense embedding-based semantic matching is widely used in e-commerce product search to address the shortcomings of lexical such as sensitivity spelling variants. The recent advances BERT-like language model encoders, have however, not found their way realtime due strict inference latency requirement imposed on websites. While bi-encoder BERT architectures enable fast approximate nearest neighbor search, training them effectively query-product data remains a challenge instabilities and persistent generalization gap with cross-encoders. In this work, we propose four-stage procedure leverage large models for while preserving low latency. We introduce interaction pre-finetuning pretrain bi-encoders improve generalization. Through offline experiments an dataset, show that distilled small BERT-based (75M params) trained using our approach improves relevance metric by up 23% over baseline DSSM-based similar only suffers 3% drop compared 20x larger teacher. also online A/B tests at scale, production exact substitute products retrieved.